import numpy as np
from sklearn.datasets import load_sample_image, load_sample_images
from skimage import io
import cv2

# Visualisation libraries

## Text
from colorama import Fore, Back, Style
from IPython.display import Image, display, Markdown, Latex

## matplotlib
import matplotlib.pyplot as plt
from matplotlib.font_manager import FontProperties
from mpl_toolkits.axes_grid1.inset_locator import inset_axes
plt.style.use('seaborn-whitegrid')
import matplotlib as mpl
mpl.rcParams['figure.figsize'] = (17, 6)

%matplotlib inline

## seaborn
import seaborn as sns

import warnings
warnings.filterwarnings("ignore")

# HTML
from IPython.core.display import HTML
HTML("""<style>.output_png {    display: table-cell;    text-align: center;    vertical-align: middle;}</style>""")

We use the sklearn image dataset to demonstrate basic operations using Open CV.

Images = load_sample_images()
Names = [x.split("\\")[-1].replace('.jpg','').title() for x in Images['filenames']]
Images = Images['images']

def ImShow(Images, Names, Title = 'Images', axis_tight = False):
    fig, ax = plt.subplots(1, 2 , figsize = (17, 6))
    ax = ax.ravel()
    font = FontProperties()
    font.set_weight('bold')
    for i in range(2):
        _ = ax[i].imshow(Images[i])
        if axis_tight:
            _ = ax[i].axis('tight')
        _ = ax[i].axis('off')
        _ = ax[i].set_title(Names[i], fontproperties=font, fontsize = 16)
    _ = fig.subplots_adjust(wspace= 0.01)
    if Title:
        _ = fig.suptitle(Title, fontproperties=font, fontsize = 18)
    return fig, ax
# Image Show
_,_ = ImShow(Images, Names, Title = 'Original Images')

Smoothing Images

2D Convolution (Image Filtering)

There are low-pass filters (LPF) and high-pass filters (HPF) that can help denoising one-dimensional signals. Typically, a LPF can remove noise by blurring the image whereas a HPF can be helpful in finding edges in an image.

In OpenCV, filter2D is a filtering function that replies on convolving a kernel with an image.

cv2.filter2D(src, ddepth, kernel[, dst[, anchor[, delta[, borderType]]]]) → dst

Parameters:

• src – input image.
• dst – output image of the same size and the same number of channels as src.

• ddepth – desired depth of the destination image; if it is negative, it will be the same as src.depth(); the following combinations of src.depth() and ddepth are supported:

  • src.depth() = CV_8U, ddepth = -1/CV_16S/CV_32F/CV_64F
  • src.depth() = CV_16U/CV_16S, ddepth = -1/CV_32F/CV_64F
  • src.depth() = CV_32F, ddepth = -1/CV_32F/CV_64F
  • src.depth() = CV_64F, ddepth = -1/CV_64F when ddepth=-1, the output image will have the same depth as the source.

• kernel – convolution kernel (or rather a correlation kernel), a single-channel floating point matrix; if you want to apply different kernels to different channels, split the image into separate color planes using split() and process them individually.

• anchor – anchor of the kernel that indicates the relative position of a filtered point within the kernel; the anchor should lie within the kernel; default value (-1,-1) means that the anchor is at the kernel center.
• delta – optional value added to the filtered pixels before storing them in dst.
• borderType – pixel extrapolation method (see borderInterpolate() for details).

To demonstrate this, we can consider the following example.

Example:

Img = Images[1].copy()

def ImgStd(Inp):
    '''This function standardizes image values.'''
    Out = np.round(Inp).astype(int)
    Out[Out<0] = 0
    Out[Out>255] = 255
    return Out

# Generating a noisy image using mu = 0 and sigma = 0.8
mu = 0; sigma = 10
Img_noise = Img.copy() - np.random.normal(mu, sigma, size=Img.shape)
Img_noise = ImgStd(Img_noise)

fig, ax = plt.subplots(1, 2 , figsize = (17, 6))
ax = ax.ravel()
font = FontProperties()
font.set_weight('bold')

_ = fig.suptitle('Flower Image', fontproperties=font, fontsize = 18)

_ = ax[0].imshow(Img)
_ = ax[0].set_title('Original Image', fontproperties=font, fontsize = 12)
_ = ax[1].imshow(Img_noise)
_ = ax[1].set_title('Noisy Image', fontproperties=font, fontsize = 12)
for i in range(2):
    _ = ax[i].set_aspect('auto')
    _ = ax[i].axis('off')
del i

# We can consider a kernal as follows
kernel = np.ones((8,8),np.float32)/64

Out = cv2.filter2D(Img_noise,-1, kernel)
Out = ImgStd(Out)

fig, ax = plt.subplots(1, 2 , figsize = (17, 6))
ax = ax.ravel()
_ = ax[0].imshow(Img_noise)
_ = ax[0].set_title('Noisy Image', fontproperties=font, fontsize = 12)
_ = ax[1].imshow(Out)
_ = ax[1].set_title('filter2D (Noisy Image)', fontproperties=font, fontsize = 12)
for i in range(2):
    _ = ax[i].set_aspect('auto')
    _ = ax[i].axis('off')
del i
del Out, kernel

Image Blurring (Image Smoothing)

Image blurring is done by removing high-frequency contents (e.g: noise, edges). There are several ways to apply this.

Blur

Box blur (also known as a box linear filter) is a form of low-pass ("blurring") filter. In a box blur, each pixel is calculated using the average value of its neighboring pixels in the input image.

Moroever, Blur is a OpenCV function that blurs an image using the normalized box filter

cv2.blur(src, ksize[, dst[, anchor[, borderType]]]) → dst

Parameters:

• src – input image; it can have any number of channels, which are processed independently, but the depth should be CV_8U, CV_16U, CV_16S, CV_32F or CV_64F.
• dst – output image of the same size and type as src.
• ksize – blurring kernel size.
• anchor – anchor point; default value Point(-1,-1) means that the anchor is at the kernel center.
• borderType – border mode used to extrapolate pixels outside of the image.

The kernel function here:

$$\texttt{K} = \frac{1}{\texttt{ksize.width*ksize.height}} \begin{bmatrix} 1 & 1 & 1 & \cdots & 1 & 1 \\ 1 & 1 & 1 & \cdots & 1 & 1 \\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\ 1 & 1 & 1 & \cdots & 1 & 1 \\ \end{bmatrix}$$

Example:

Out = cv2.blur(Img_noise,(8,8))

fig, ax = plt.subplots(1, 2 , figsize = (17, 6))
ax = ax.ravel()
_ = ax[0].imshow(Img_noise)
_ = ax[0].set_title('Noisy Image', fontproperties=font, fontsize = 12)
_ = ax[1].imshow(Out)
_ = ax[1].set_title('blur (Noisy Image)', fontproperties=font, fontsize = 12)
for i in range(2):
    _ = ax[i].set_aspect('auto')
    _ = ax[i].axis('off')
del i
del Out

To see this filter better, we can consider using an image with some text phrases.

Example:

Colors={'White':(255, 255, 255), 'Black':(0, 0, 0), 'Red':(255,0,0),'Green':(0,255,0), 'Blue':(0,0,255),
        'Yellow':(255,255,0), 'Purple':(128,0,128), 'Indigo':(75,0,130),'OrangeRed':(255,69,0)}

Img2 = np.zeros((800, 800, 3), np.uint8)

Img2 = cv2.putText(Img2, text = 'Sample Text', org = (80, 300),
                  fontFace = cv2.FONT_HERSHEY_TRIPLEX, fontScale = 3, color = Colors['Yellow'], thickness = 2)

Img2 = cv2.putText(Img2, text = 'Sample Text', org = (80, 400),
                  fontFace = cv2.FONT_HERSHEY_TRIPLEX, fontScale = 3, color = Colors['Red'], thickness = 2)

Img2 = cv2.putText(Img2, text = 'Sample Text', org = (80, 500),
                  fontFace = cv2.FONT_HERSHEY_TRIPLEX, fontScale = 3, color = Colors['Green'], thickness = 2)

# Generating a noisy image using mu = 0 and sigma = 0.8
mu = 0; sigma = 50
Img2_noise = Img2.copy() - np.random.normal(mu, sigma, size=Img2.shape)
Img2_noise = ImgStd(Img2_noise)


fig, ax = plt.subplots(1, 2 , figsize = (17, 6))
ax = ax.ravel()
_ = fig.suptitle('Image with Texts', fontproperties=font, fontsize = 18)

_ = ax[0].imshow(Img2)
_ = ax[0].set_title('Original Image', fontproperties=font, fontsize = 12)
_ = ax[1].imshow(Img2_noise)
_ = ax[1].set_title('Noisy Image', fontproperties=font, fontsize = 12)
for i in range(2):
    _ = ax[i].set_aspect('auto')
    _ = ax[i].axis('off')
del i

Out = cv2.blur(Img2_noise,(8,8))

fig, ax = plt.subplots(1, 2 , figsize = (17, 6))
ax = ax.ravel()
_ = ax[0].imshow(Img2_noise)
_ = ax[0].set_title('Noisy Image', fontproperties=font, fontsize = 12)
_ = ax[1].imshow(Out)
_ = ax[1].set_title('blur (Noisy Image)', fontproperties=font, fontsize = 12)
for i in range(2):
    _ = ax[i].set_aspect('auto')
    _ = ax[i].axis('off')
del i
del Out

Box Filter

Box Filter is a OpenCV function that blurs an image using the box filter.

cv2.boxFilter(src, ddepth, ksize[, dst[, anchor[, normalize[, borderType]]]]) → dst

Parameters:

• src – input image.
• dst – output image of the same size and type as src.
• ddepth – the output image depth (-1 to use src.depth()).
• ksize – blurring kernel size.
• anchor – anchor point; default value Point(-1,-1) means that the anchor is at the kernel center.
• normalize – flag, specifying whether the kernel is normalized by its area or not.
• borderType – border mode used to extrapolate pixels outside of the image.

The function smoothes an image using the kernel:

$$\texttt{K} = \alpha \begin{bmatrix} 1 & 1 & 1 & \cdots & 1 & 1 \\ 1 & 1 & 1 & \cdots & 1 & 1 \\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\ 1 & 1 & 1 & \cdots & 1 & 1 \\ \end{bmatrix}$$

where

$$\alpha = \begin{cases} \frac{1}{\texttt{ksize.width*ksize.height}}, & \mbox{when normalize=true},\\ 1, & \mbox{Otherwise}. \end{cases}$$

Example:

fig, ax = plt.subplots(2, 2 , figsize = (17, 2*6))
ax = ax.ravel()
_ = ax[0].imshow(Img)
_ = ax[0].set_title('Original Image', fontproperties=font, fontsize = 12)

for i in range(1,4):
    k = (2**(i-1))*10+1
    _ = ax[i].imshow(cv2.boxFilter(src = Img, ddepth = -1, ksize = (k, k)))
    _ = ax[i].set_title('boxFilter (Image)', fontproperties=font, fontsize = 12)
    _ = ax[i].text(x = 0, y = int(0.02*Img.shape[1]), s = 'Kernel Size: %i by %i' % (k, k),
                   size=11, color = 'Navy', bbox=dict(boxstyle="square", ec='Navy',fc='LightSkyBlue'))
    
for i in range(len(ax)):
    _ = ax[i].set_aspect('auto')
    _ = ax[i].axis('off')
del i, k

Gaussian Blur

Gaussian function in one dimension can be defined as $$G(x)={\frac {1}{\sqrt {2\pi \sigma ^{2}}}}e^{-{\frac {x^{2}}{2\sigma ^{2}}}}$$ In two dimensions, it is the product of two such Gaussian functions, one in each dimension: $$G(x,y)={\frac {1}{2\pi \sigma ^{2}}}e^{-{\frac {x^{2}+y^{2}}{2\sigma ^{2}}}}$$ where x is the distance from the origin in the horizontal axis, y is the distance from the origin in the vertical axis, and σ is the standard deviation of the Gaussian distribution.

In OpenCV, Gaussian Blur is defined as follows.

cv2.GaussianBlur(src, ksize, sigmaX[, dst[, sigmaY[, borderType]]]) → dst

Parameters:

• src – input image; the image can have any number of channels, which are processed independently, but the depth should be CV_8U, CV_16U, CV_16S, CV_32F or CV_64F.
• dst – output image of the same size and type as src.
• ksize – Gaussian kernel size. ksize.width and ksize.height can differ but they both must be positive and odd. Or, they can be zero’s and then they are computed from sigma* .
• sigmaX – Gaussian kernel standard deviation in X direction.
• sigmaY – Gaussian kernel standard deviation in Y direction; if sigmaY is zero, it is set to be equal to sigmaX, if both sigmas are zeros, they are computed from ksize.width and ksize.height , respectively (see getGaussianKernel() for details); to fully control the result regardless of possible future modifications of all this semantics, it is recommended to specify all of ksize, sigmaX, and sigmaY.
• borderType – pixel extrapolation method (see borderInterpolate() for details).

Example:

fig, ax = plt.subplots(2, 2 , figsize = (17, 2*6))
ax = ax.ravel()
_ = ax[0].imshow(Img)
_ = ax[0].set_title('Original Image', fontproperties=font, fontsize = 12)

for i in range(1,4):
    k = (2**(i-1))*4+1
    _ = ax[i].imshow(cv2.GaussianBlur(src = Img, ksize = (k, k), sigmaX = 0))
    _ = ax[i].set_title('GaussianBlur (Image)', fontproperties=font, fontsize = 12)
    _ = ax[i].text(x = 0, y = int(0.02*Img.shape[1]), s = 'Kernel Size: %i by %i' % (k, k),
                   size=11, color = 'Navy', bbox=dict(boxstyle="square", ec='Navy',fc='LightSkyBlue'))

for i in range(len(ax)):
    _ = ax[i].set_aspect('auto')
    _ = ax[i].axis('off')
del i, k

Median Blur

The median filter is often used to remove noise from an image that preserves edges while removing noise. See A Fast Two-Dimensional Median Filtering Algorithm by Thomas S. Huang et al. for more details regarding this filter.

Median Blur is a OpenCV function that blurs an image using the median filter.

cv2.medianBlur(src, ksize[, dst]) → dst

Parameters:

• src – input 1-, 3-, or 4-channel image; when ksize is 3 or 5, the image depth should be CV_8U, CV_16U, or CV_32F, for larger aperture sizes, it can only be CV_8U.
• dst – destination array of the same size and type as src.
• ksize – aperture linear size; it must be odd and greater than 1, for example: 3, 5, 7 ...

Example:

fig, ax = plt.subplots(2, 2 , figsize = (17, 2*6))
ax = ax.ravel()
_ = ax[0].imshow(Img)
_ = ax[0].set_title('Original Image', fontproperties=font, fontsize = 12)

for i in range(1,4):
    k = (2**(i-1))*10+1
    _ = ax[i].imshow(cv2.medianBlur(Img, k, cv2.BORDER_DEFAULT))
    _ = ax[i].set_title('medianBlur (Image)', fontproperties=font, fontsize = 12)
    _ = ax[i].text(x = 0, y = int(0.02*Img.shape[1]), s = 'Kernel Size: %i by %i' % (k, k),
                   size=11, color = 'Navy', bbox=dict(boxstyle="square", ec='Navy',fc='LightSkyBlue'))
for i in range(len(ax)):
    _ = ax[i].set_aspect('auto')
    _ = ax[i].axis('off')
del i, k

Bilateral Filtering

A bilateral filter is a non-linear, edge-preserving, and noise-reducing smoothing filter for images. The bilateral filter is defined as $$I^{\text{filtered}}(x)={\frac {1}{W_{p}}}\sum _{x_{i}\in \Omega }I(x_{i})f_{r}(\|I(x_{i})-I(x)\|)g_{s}(\|x_{i}-x\|).$$

where

• $I^\text{filtered}$ is the filtered image,
• $I$ is the original input image to be filtered,
• $x$ are the coordinates of the current pixel to be filtered,
• $\Omega$ is the window centered in $x$, so $x_{i}\in \Omega$ is another pixel,
• $f_r$ is the range kernel for smoothing differences in intensities (this function can be a Gaussian function),
• $g_{s}$ is the spatial (or domain) kernel for smoothing differences in coordinates (this function can be a Gaussian function).
• $W_{p}$ is the normalization term, and it is defined as follows $$\displaystyle W_{p}=\sum _{x_{i}\in \Omega }{f_{r}(\|I(x_{i})-I(x)\|)g_{s}(\|x_{i}-x\|)}$$

See Bilateral Filtering for Gray and Color Images for more details about Bilateral Filtering.

In OpenCV, Bilateral Filtering is a function that applies the bilateral filter to an image.

cv2.bilateralFilter(src, d, sigmaColor, sigmaSpace[, dst[, borderType]]) → dst

Parameters:

• src – Source 8-bit or floating-point, 1-channel or 3-channel image.
• dst – Destination image of the same size and type as src .
• d – Diameter of each pixel neighborhood that is used during filtering. If it is non-positive, it is computed from sigmaSpace .
• sigmaColor – Filter sigma in the color space. A larger value of the parameter means that farther colors within the pixel neighborhood (see sigmaSpace ) will be mixed together, resulting in larger areas of semi-equal color.
• sigmaSpace – Filter sigma in the coordinate space. A larger value of the parameter means that farther pixels will influence each other as long as their colors are close enough (see sigmaColor ). When d>0 , it specifies the neighborhood size regardless of sigmaSpace . Otherwise, d is proportional to sigmaSpace .

Example:

fig, ax = plt.subplots(2, 2 , figsize = (17, 2*6))
ax = ax.ravel()
_ = ax[0].imshow(Img)
_ = ax[0].set_title('Original Image', fontproperties=font, fontsize = 12)

for i in range(1,4):
    k = (2**(i-1))*10+1
    _ = ax[i].imshow(cv2.bilateralFilter(src = Img, d = k, sigmaColor = 75, sigmaSpace = 75))
    _ = ax[i].set_title('bilateralFilter (Image)', fontproperties=font, fontsize = 12)
    _ = ax[i].text(x = 0, y = int(0.02*Img.shape[1]), s = 'Diameter = %i' % k,
                   size=11, color = 'Navy', bbox=dict(boxstyle="square", ec='Navy',fc='LightSkyBlue'))
for i in range(len(ax)):
    _ = ax[i].set_aspect('auto')
    _ = ax[i].axis('off')
del i, k

Moreover, for adaptive bilateral filter, see this article.

---

Refrences

1. OpenCV documentation
2. Smoothing Images
3. Box Blur
4. Gaussian blur
5. Bilateral filter
6. Huang, T., G. J. T. G. Y. Yang, and Greory Tang. "A fast two-dimensional median filtering algorithm." IEEE Transactions on Acoustics, Speech, and Signal Processing 27.1 (1979): 13-18.
7. Gavaskar, Ruturaj G., and Kunal N. Chaudhury. "Fast adaptive bilateral filtering." IEEE Transactions on Image Processing 28.2 (2018): 779-790.

---